Support for Azure storage by tramora · Pull Request #571 · KhiopsML/khiops-python

tramora · 2026-04-10T11:36:53Z

Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects)
The only supported authentication method is currently the AZURE_STORAGE_CONNECTION_STRING embedding the account name and account key

Completes #244 and fixes #576

TODO Before Asking for a Review

Rebase your branch to the latest version of main (or main-v10)
Make sure all CI workflows are green
When adding a public feature/fix: Update the Unreleased section of CHANGELOG.md (no date)
Self-Review: Review "Files Changed" tab and fix any problems you find
API Docs (only if there are changes in docstrings, rst files or samples):
- Check the docs build without warning: see the log of the API Docs workflow
- Check that your changes render well in HTML: download the API Docs artifact and open index.html
- If there are any problems it is faster to iterate by building locally the API Docs

tramora · 2026-04-13T08:49:36Z

The automated tests on conda environments currently fail because of an outdate version of the azure-storage-file-share package in the conda-forge channel. This PR conda-forge/azure-storage-file-share-feedstock#9 should fix the issue but it is blocked for the moment.

folmos-at-orange

I think this is an opportunity to have better docs on file system.

Currently, very little is documented, and we now have 3 different remote systems of which only one (S3) is well documented.

I'd a section for each remote system in the module's docstring. Each section contains how to configure it in each case, and a URI example.

popescu-v · 2026-04-29T17:55:48Z

Check the remaining comments. Check failing remote tests in https://github.com/KhiopsML/khiops-python/actions/runs/24792699160, e.g. https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:279 and https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:294.

The error message was not very explicit; khiops failed only with a fatal error : Command file : Batch mode failure.

Naturally the latest ubuntu dev image didn't have the azure driver (system-wide or Conda) as the current PR is not yet merged.

The latest Conda packages for GCS (0.0.16) and Azure (0.0.6) do not exist (compared to their counterpart on github). And the previous versions are not installable under Conda because a (new) dependency incompatibility (fmt).

Anyway the khiopspydev-ubuntu22.04:11.0.0.244-support-azure-storage image contains the system-wide drivers that should be always be the fallback drivers during the executions for both "pyXX" (simulated venv) and "pyXX_conda" (Conda environment). However despite of this, the execution in a Conda environment fails. I tried to launch locally the container as described in khiops-core#1009

Noted. However:

the native drivers are not the fallback drivers in Conda environments, because the drivers are selected according to the value of the KHIOPS_DRIVERS_PATH environment variable as defined in khiops_env. Its value is specific to the Conda environment khiops is launched in;
however, the error message in case the driver is missing could be more helpful IMHO, rather than just stating that the file is not found; I'd file an issue in the khiops repo to this end.

popescu-v

A few remaining comments.

tramora · 2026-04-30T09:09:53Z

Check the remaining comments. Check failing remote tests in https://github.com/KhiopsML/khiops-python/actions/runs/24792699160, e.g. https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:279 and https://github.com/KhiopsML/khiops-python/actions/runs/24792699160/job/72554199345#step:12:294.

The error message was not very explicit; khiops failed only with a fatal error : Command file : Batch mode failure.
Naturally the latest ubuntu dev image didn't have the azure driver (system-wide or Conda) as the current PR is not yet merged.
The latest Conda packages for GCS (0.0.16) and Azure (0.0.6) do not exist (compared to their counterpart on github). And the previous versions are not installable under Conda because a (new) dependency incompatibility (fmt).
Anyway the khiopspydev-ubuntu22.04:11.0.0.244-support-azure-storage image contains the system-wide drivers that should be always be the fallback drivers during the executions for both "pyXX" (simulated venv) and "pyXX_conda" (Conda environment). However despite of this, the execution in a Conda environment fails. I tried to launch locally the container as described in khiops-core#1009

Noted. However:
* the native drivers are not the fallback drivers in Conda environments, because the drivers are selected according to the value of the `KHIOPS_DRIVERS_PATH` environment variable as defined in `khiops_env`. Its value is specific to the Conda environment `khiops` is launched in;

* however, the error message in case the driver is missing could be more helpful IMHO, rather than just stating that the file is not found; I'd file an issue in the `khiops` repo to this end.

When the conditions described in the issue khiops-core#1009 are not met (which make the process fail far before), khiops-core is explicit enough in telling it cannot open the remote file. Thus there is no need opening a new issue.

- Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects) - The only supported authentication method is currently the `AZURE_STORAGE_CONNECTION_STRING` embedding the account name and account key

…cess.py

- Remove the environments creation in the docker files - Remove the installation of the Conda packages - Remove the usage of these environments in the tests (unit and integration tests)

popescu-v

LGTM.
Update Azure driver version and remove # XXX comments accordingly in the testing infrastructure when this becomes possible.

tramora force-pushed the 244-support-azure-storage branch 3 times, most recently from be97119 to 08f91e2 Compare April 10, 2026 14:34

tramora requested review from folmos-at-orange and popescu-v April 10, 2026 15:00

tramora force-pushed the 244-support-azure-storage branch 4 times, most recently from 17fe956 to 80691c3 Compare April 12, 2026 12:59

tramora force-pushed the 244-support-azure-storage branch from 80691c3 to 701678b Compare April 13, 2026 16:32

folmos-at-orange requested changes Apr 14, 2026

View reviewed changes

tramora force-pushed the 244-support-azure-storage branch from 701678b to 19aa231 Compare April 15, 2026 16:39

tramora requested a review from folmos-at-orange April 15, 2026 16:47

folmos-at-orange approved these changes Apr 16, 2026

View reviewed changes

Comment thread khiops/core/internals/filesystems.py Outdated

Comment thread khiops/core/internals/filesystems.py Outdated

Comment thread khiops/core/internals/filesystems.py Outdated

Comment thread khiops/core/internals/filesystems.py

tramora force-pushed the 244-support-azure-storage branch from 19aa231 to 4dc0953 Compare April 16, 2026 14:39